Method and device for supplying radar data

ABSTRACT

A computer-implemented method for supplying radar data. The method includes the following steps: receiving input data, the input data including satellite images; generating radar data using a trained machine learning algorithm, which is applied to the input data; and outputting the generated radar data.

CROSS REFERENCE

The present application claims the benefit under 35 U.S.C. § 119 ofGerman Patent Application No. DE 102019216607.5 filed on Oct. 29, 2019,which is expressly incorporated herein by reference in its entirety.

FIELD

The present invention relates to a computer-implemented method forsupplying radar data, to a device for supplying radar data, a computerprogram and a non-volatile, computer-readable memory medium.

BACKGROUND INFORMATION

In the field of automated driving, the precise localization of thevehicle is required, which may be used for estimating the positions ofthe components of the vehicle. Radar data are able to be used for thelocalization. The radar data may encompass dense point clouds, which aregenerated with the aid of radar measurements. Vehicles can drive throughthe regions to be detected and radar maps are generated based on radarmeasurements of the vehicles. However, the generation of the radar mapconsumes considerable time because all roads in the region to bedetected have to be traveled. The generation of the radar maps is alsoexpensive because considerable expense may arise for vehicles, fuel anddrivers.

A localization based on 3D maps is described in Carle et al., “GlobalRover Localization by Matching Lidar and Orbital 3D Maps,” IEEEInternational Conference on Robotics and Automation, 2010.

A method for improving existing maps is described in Vysotska et al.,“Exploiting Building Information from Publicly Available Maps inGraph-Based SLAM,” IEEE/RSJ International Conference on IntelligentRobots and Systems, 2016.

A method for road segmentation is described in Wang et al., “A Review ofRoad Extraction from Remote Sensing Images,” Journal of Traffic andTransportation Engineering, 3(3), 271-282, 2016. A further such methodis described in Mnih et al., “Learning to Detect Roads inHigh-Resolution Aerial Images,” European Conference on Computer Vision(ECCV), 2010.

A method based on neural networks is described in Xia et al. “RoadExtraction from High Resolution Image with Deep Convolution Network—ACase Study of GF-2 Image,” International Electronic Conference on RemoteSensing, 2018.

SUMMARY

According to a first aspect, the present invention relates to acomputer-implemented method for supplying radar data. Input data whichinclude satellite images are received. With the aid of a trained machinelearning algorithm, which is applied to the input data, radar data aregenerated. The generated radar data are output.

According to a second aspect, the present invention relates to a devicefor supplying radar data, which has an input interface, a processingunit, and an output interface. The input interface is designed toreceive input data, the input data including satellite images. Theprocessing unit is designed to generate radar data using a trainedmachine learning algorithm, which is applied to the input data. Theoutput interface is designed to output the generated radar data.

According to a third aspect, the present invention relates to a computerprogram which when executed on a computer, induces the computer tocontrol the execution of the steps of the computer-implemented methodaccording to the first aspect.

According to a fourth aspect, the present invention relates to anon-volatile, computer-readable memory medium, which stores anexecutable computer program which when executed on a computer, inducesthe computer to control the execution of the steps of thecomputer-implemented method according to the first aspect.

Preferred embodiments of the present invention are described herein.

Satellite images are well suited to detecting large regions. Moreover,satellite images are typically available at a reasonable cost. Thepresent invention utilizes the fact that certain structures are easilydetectable both in satellite images and in radar measurements. Bothmethods in particular detect road structures such as posts, guardrailsor traffic signs very well. This allows for a computer-implementedtransmission of the satellite images in radar data.

In addition, the radar data are able to be generated without an undueinvestment in time because no travel along the roads in the region to bedetected is required.

According to the present invention, ‘generated radar data’ should beunderstood as synthetic radar data, which were generated with the aid ofthe satellite images. These should correspond as closely as possible toreal radar data generated by radar measurements.

According to one specific embodiment of the computer-implemented methodin accordance with the present invention, radar data are generated byradar sensors of a motor vehicle. The radar data generated with the aidof the radar sensors of the motor vehicle are compared to the radar datagenerated based on the satellite images. The motor vehicle is located onthe basis of the comparison. Excellent scalability is advantageous inthis context. If satellite images are available in a region, then thelocalization there is able to be carried out directly. This allows for alocalization in many regions.

According to one specific embodiment of the computer-implemented methodin accordance with the present invention, the application of the trainedmachine learning algorithm to the input data includes a semanticsegmentation of the satellite images. In particular certain structuressuch as traffic lane markings, traffic signs, guardrails and similarthings are able to be automatically detected.

According to one specific embodiment of the computer-implemented methodin accordance with the present invention, radar segment images in whichradar measurements are to be expected are generated with the aid of thesemantic segmentation. The machine learning algorithm allocates valuesfor the radar cross section (RCS) to the pixels in the radar segmentimages.

According to a specific embodiment of the computer-implemented method inaccordance with the present invention, the pixels of the satelliteimages are two-dimensional at the outset. The values for the radar crosssection allocated to the pixels are transformed in point clouds of radarmeasurements in a three-dimensional world coordinate system.

According to a specific embodiment of the computer-implemented method inaccordance with the present invention, altitude information is takeninto account when the two-dimensional coordinates are transformed intothe three-dimensional coordinates.

According to a specific embodiment of the computer-implemented method inaccordance with the present invention, the generated radar data includepoint clouds. Alternatively or additionally, the generated radar datamay encompass Gaussian distributions.

According to a specific embodiment of the computer-implemented method inaccordance with the present invention, radar maps are produced by anextraction of features with the aid of the point clouds and/or Gaussiandistributions.

According to a specific embodiment of the computer-implemented method inaccordance with the present invention, the machine learning algorithm istrained by monitored learning. In particular, the machine learningalgorithm may be based on a deep learning model for semanticsegmentation.

According to a specific embodiment of the computer-implemented method inaccordance with the present invention, the training of the machinelearning algorithm by monitored learning takes place with the aid oftraining data, the training data including satellite images as inputdata and real radar data as output data. The real radar data correspondto radar segment images that correspond to projections of the values ofradar cross sections of real radar measurements. The radar data acquiredin the real radar measurements are typically available in a radarcoordinate system. The radar data are transformed from the radarcoordinate system into a world coordinate system. This makes it possibleto image the radar data directly onto the satellite images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic block diagram of a device for supplying radardata according to a specific embodiment of the present invention.

FIG. 2 shows a schematic block diagram in order to describe thelocalization of a motor vehicle with the aid of radar data that weregenerated on the basis of satellite images in accordance with an exampleembodiment of the present invention.

FIG. 3 shows a schematic block diagram in order to describe thegeneration of radar data with the aid of satellite images in accordancewith an example embodiment of the present invention.

FIG. 4 shows a flow diagram of a computer-implemented method forsupplying radar data according to a specific embodiment of the presentinvention in accordance with an example embodiment of the presentinvention.

FIG. 5 shows a schematic block diagram of a computer program accordingto a specific embodiment of the present invention.

FIG. 6 shows a schematic block diagram of a non-volatile,computer-readable memory medium according to a specific embodiment ofthe present invention.

The numbering of method steps is provided for reasons of clarity and isgenerally not meant to imply a certain sequence in time. It isparticularly also possible to carry out multiple method stepssimultaneously.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1 shows a schematic block diagram of a device 1 for supplying radardata according to a specific embodiment of the present invention. Device1 includes an input interface 11 for the supply of input data. Via acable connection or a wireless connection, the input interface is ableto be connected to at least one external device, in particular to asatellite system or a server in order to receive the input data. Theinput data include satellite images. Satellite images may be understoodboth as a single satellite image and a multitude of satellite images.The input data may include additional information such as altitudeinformation or semantic labels of the satellite images.

Device 1 furthermore has a memory 12 in which the received input dataare stored. Further data, which are required in order to execute atrained machine learning algorithm, are able to be stored in memory 12.

In addition, device 1 has a processing unit 13, which is designed toexecute the trained machine learning algorithm. Processing unit 13 mayinclude at least one of the following: processors, microprocessors,integrated circuits, ASICs and the like. Processing unit 13 accesses theinput data stored in memory 12. Using the trained machine learningalgorithm, processing unit 13 detects segments in the satellite imagesin which radar measurements may occur. The segments thus correspond toobjects or structures at which the radar beams are reflected. Processingunit 13 is able to identify pixels in the satellite images thatcorrespond to these segments. The pixels correspond to spatial positionsat which radar reflections may occur.

The machine learning algorithm may have been trained with the aid oftraining data that include satellite images as input data as well asreal radar data as output data. Processing unit 13 may be designed tocarry out the training of the machine learning algorithm on its own. Asan alternative, it is also possible that an already trained machinelearning algorithm is made available.

Processing unit 13 is able to allocate values for a radar cross sectionto the pixels corresponding to spatial positions in which radarreflections may occur. The values are also able to be generated with theaid of the trained machine learning algorithm.

In addition, processing unit 13 may be designed to transform the valuesfor the radar reflections allocated to the two-dimensional pixels intothree-dimensional point clouds, for which altitude information is ableto be taken into account.

Finally, processing unit 13 may be designed to produce radar maps withthe aid of the three-dimensional point clouds, features being able to beextracted.

The radar maps are able to be supplied in any representational manner.For example, instead of point clouds or in addition to the point clouds,Gaussian distributions may be generated.

Input interface 11 is furthermore able to receive real radar data thatare generated by radar sensors of a motor vehicle. Processing unit 13 isdesigned to compare the received real radar data to the synthetic radardata generated with the aid of the satellite images. Based on thecomparison, e.g., by registration of the real radar data and thesynthetic radar data, it is possible to locate the motor vehicle.Processing unit 13 is able to output the position of the motor vehicle.In particular, driver assistance systems are able to control functionsof the motor vehicle based on the localization of the motor vehicle.

In addition, device 1 has an output interface 14 for the output of thesynthetic radar data or for the localization of the motor vehicle.Output interface 14 may be identical with input interface 11.

FIG. 2 shows a schematic block diagram in order to describe thelocalization of a motor vehicle with the aid of radar data generatedbased on satellite images. Satellite images 21 are made available. Atransformation algorithm 22, which generates synthetic radar data 23, isapplied to satellite images 21. The transformation algorithm is based onthe afore-described trained machine learning algorithm. Real radar data24, which are generated by radar sensors of a motor vehicle, areadditionally made available. With the aid of a comparison, alocalization 25 of the motor vehicle is performed. Pose 26 of the motorvehicle is able to be ascertained based on the localization of the motorvehicle.

FIG. 3 shows a schematic block diagram in order to describe thegeneration of radar data with the aid of satellite images. A satelliteimage 31 is supplied in which certain structures such as traffic lanedemarcations can be seen. Through a semantic segmentation, pixels andsegments corresponding to structures that reflect radar beams areidentified with the aid of the trained machine learning algorithm.Moreover, values for the radar cross section are allocated to thepixels. From this, pixel clouds 33 are generated. Radar maps 34 areproduced by the extraction of features.

FIG. 4 shows a flow diagram of a computer-implemented method forsupplying radar data according to a specific embodiment of the presentinvention. The method is able to be carried out by an afore-describeddevice 1. Conversely, device 1 may be developed to carry out the methoddescribed in the following text.

In a first method step S1, input data which encompass satellite imagesare received.

In a method step S2, a machine learning algorithm is trained. Certainsatellite images are supplied as input data for this purpose and radardata corresponding to the satellite images are supplied as output datain order to carry out monitored learning. The machine learning algorithmtrained in this manner is then able to be applied to a wide variety ofsatellite images. The radar data for the training are optionally able tobe prepared. For example, an annotation may be made by a user. It mayalso be provided to implement an automatic annotation. An excellentglobal localization of the radar data is thereby able to be achieved ata reduced work investment.

In a method step S3, synthetic radar data are generated by applying thetrained machine learning algorithm to the input data. Toward this end,radar segments or pixels which correspond to objects from which radarradiation is reflected may first be identified. In addition, values forthe radar cross section are indicated for the pixels. With the aid ofaltitude information, for example, three-dimensional radar data are ableto be generated such as in the form of point clouds and/or Gaussiandistributions. The extraction of features moreover makes it possible toproduce radar maps that additionally include information pertaining tocertain structures.

In a method S4, the generated radar data are output.

The present method may furthermore be used for locating a motor vehicle.Toward this end, radar sensors of the motor vehicle generate real radardata in a fifth method step S5.

In a method step S6, the real radar data generated with the aid of theradar sensors are compared to the synthetic radar data generated withthe aid of the satellite images. In particular, a registration may becarried out, i.e. the real radar data are rotated and shifted in such away that they agree or coincide as closely as possible with thesynthetic radar data.

This makes it possible to locate the motor vehicle in a further methodstep S7. A pose of the motor vehicle, in particular, is able to becalculated. Certain driving functions may be automatically controlledusing the localization of the motor vehicle.

FIG. 5 shows a schematic block diagram of a computer program 5 accordingto a specific embodiment of the present invention. Computer program 5includes executable program code 51, which when executed on a computerinduces the computer to control and/or carry out the afore-described,computer-implemented method for supplying radar data.

FIG. 6 shows a schematic block diagram of a non-volatile,computer-readable memory medium 6 according to a specific embodiment ofthe present invention. Memory medium 6 includes executable program code61, which when executed on a computer induces the computer to controland/or carry out the afore-described computer-implemented method forsupplying radar data.

What is claimed is:
 1. A computer-implemented method for supplying radardata, comprising the following steps: receiving input data, the inputdata including satellite images; generating radar data using a trainedmachine learning algorithm, which is applied to the input data; andoutputting the generated radar data.
 2. The method as recited in claim1, further comprising the following steps: generating radar data byradar sensors of a motor vehicle; comparing the radar data generated bythe radar sensors of the motor vehicle to the radar data generated usingthe satellite images; and locating the motor vehicle based on thecomparison.
 3. The method as recited in claim 1, wherein the applicationof the trained machine learning algorithm to the input data includes asemantic segmentation of the satellite images.
 4. The method as recitedin claim 1, wherein the generated radar data include point clouds and/orGaussian distributions.
 5. The method as recited in claim 4, furthercomprising the following step: producing radar maps by an extraction offeatures using the point clouds and/or the Gaussian distributions. 6.The method as recited in claim 1, further comprising the following step:training the machine learning algorithm by monitored learning.
 7. Themethod as recited in claim 6, wherein the training of the machinelearning algorithm by monitored learning takes place based on trainingdata, the training data including satellite images as input data andreal radar data as output data.
 8. A device for supplying radar data,comprising: an input interface configured to receive input data, theinput data including satellite images; a processing unit configured togenerate radar data using a trained machine learning algorithm, which isapplied to the input data; and an output interface configured to outputthe generated radar data.
 9. A non-transitory, non-volatile,computer-readable memory medium on which is stored a computer programfor supplying radar data, the computer program, when executed by acomputer, causing the computer to perform the following steps: receivinginput data, the input data including satellite images; generating radardata using a trained machine learning algorithm, which is applied to theinput data; and outputting the generated radar data.